feat(fuse): add hash-distributed partition writes - #20216
Conversation
113b6c9 to
1f7e778
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3f2eb9d4fd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7789204554
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
bucket(count, value)partition transform using SipHash.WRITE_DISTRIBUTION_MODE = 'hash'for partitioned Fuse tables.TableWritePrepare, evaluate the partition expressions, and route equal partition values throughGlobalShufflebefore distributed INSERT SELECT writers.table.append_data(): evaluate partition expressions, full-sort by partition values, and rebuild blocks using the table'smax_rows_per_block.The goal is to reduce fragmented blocks when unordered input repeatedly crosses physical partition boundaries, while preserving the existing guarantee that a block or segment never spans physical partitions.
Current scope
GlobalShuffleto route equal evaluated partition values to the same writer node.WRITE_DISTRIBUTION_MODE = 'hash'keep their existing write behavior.bucket(count, value)controls persistent physical partition cardinality. It does not configure execution nodes or pipeline lanes.Tests
Type of change
AI assistance
This change is